Tomáš Pospíšek's Notizblock

drop open request from

tldr: sysctl net.ipv4.tcp_max_syn_backlog $higher_number

Our monitoring system was reporting "socket timeout after 10 seconds" to some of our websites.

Further investigation lead to these entries in /var/log/debug:

Sep 30 06:54:00 webserver kernel: [22536160.488069] TCP: drop open request from 10.10.10.10/1234

There's a lot of stuff to be found on the internet about "drop open request from", but none seems to be very to the point.

So let's dive into the problem and first have a look where the message "drop open request from" comes from:

It gets logged by the pr_drop_req function in the kernel.

The pr_drop_req function is called exclusively here.

As one can see, the code there:

if (!net->ipv4.sysctl_tcp_syncookies &&
            (net->ipv4.sysctl_max_syn_backlog - inet_csk_reqsk_queue_len(sk) <
             (net->ipv4.sysctl_max_syn_backlog >> 2)) &&
            !tcp_peer_is_proven(req, dst)) {
            [...]
            pr_drop_req(req, ntohs(tcp_hdr(skb)->source), rsk_ops->family);

is checking whether:

• syncookies are not being used
• sysctl_max_syn_backlog is full
• and a third condition that I choose to ignore here

If all conditions are true, then the packet is dropped.

As far as I understand the "syn_backlog" contains half-open TCP connections, which means connections for which the receiver has seen a SYN packet, but the whole SYN, SYN-ACK, ACK process of establishing a working TCP connection has not yet been completed.

What the kernel seems to be trying to do here is to make sure that a DoS attacker or some broken network equipment can not fill the available system ressources with TCP connection open requests, which would never be completed and thus would hog the buffer for TCP connections that are being established forever.

The system administrator can work around the "drop open request from" problem in various ways:

• make sure that syn cookes are enabled - this is somewhat documented here
• make sure that the "syn_backlog" is not being filled up:
  • either by not filling it up in the first place or
  • by increasing the respective buffer

I've taken the latter aproach: on the Debian system that was dropping TCP connections sysctl_max_syn_backlog was set to a mere 128. I've increased that to 1024:

sysctl net.ipv4.tcp_max_syn_backlog 1024

and the problems went away. Some people on the internet are increasing that number much, much, much more.

Tomáš Pospíšek, 2018-10-07